2. Weight Initialization

Weights need initial values before training starts.

Poor initialization can cause:

• Vanishing gradients
• Exploding gradients
• Slow or unstable training

Common Initialization Techniques

Xavier/Glorot	Commonly used with Sigmoid/Tanh
He/Kaiming	Commonly used with ReLU

He Initialization

Designed for ReLU networks to keep activations from becoming too small or large.

nn.init.kaiming_normal_(model.linear1.weight, nonlinearity="relu")

Xavier Initialization

Commonly used with Sigmoid or Tanh activations.

nn.init.xavier_normal_(model.linear1.weight)

Actual Implementation Full code

Code Block

import torch
import torch.nn as nn


class NeuralNetwork(nn.Module):
    def __init__(self):
        super().__init__()


        self.linear1 = nn.Linear(2, 10)
        self.linear2 = nn.Linear(10, 1)


        # He Initialization for ReLU
        nn.init.kaiming_normal_(
            self.linear1.weight,
            nonlinearity="relu"
        )


        nn.init.zeros_(self.linear1.bias)


        # Xavier Initialization
        nn.init.xavier_normal_(self.linear2.weight)
        nn.init.zeros_(self.linear2.bias)


    def forwardpass(self, x):
        x = torch.relu(self.linear1(x))
        x = self.linear2(x)
        return x


neuralnetwork = NeuralNetwork()

import torch import torch.nn as nn class NeuralNetwork(nn.Module): def __init__(self): super().__init__() self.linear1 = nn.Linear(2, 10) self.linear2 = nn.Linear(10, 1) # He Initialization for ReLU nn.init.kaiming_normal_( self.linear1.weight, nonlinearity="relu" ) nn.init.zeros_(self.linear1.bias) # Xavier Initialization nn.init.xavier_normal_(self.linear2.weight) nn.init.zeros_(self.linear2.bias) def forwardpass(self, x): x = torch.relu(self.linear1(x)) x = self.linear2(x) return x neuralnetwork = NeuralNetwork()